In a Linux or UNIX operating system (OS), everything that accepts or sends data to or from the OS is considered to be a device. Common SCSI devices include terminals, hard disks, printers, CD drives, and modems. A rule of thumb is that anything used for input, storage, or handling of data in an OS is considered a device and needs a device entry so that it may be detected by the OS.
Each of these devices may also be SCSI devices. SCSI (small computer system interface) is a family of standards that connects components via a parallel bus for system-level interfacing between a computer and intelligent devices such as hard disks, floppy disks, CD ROMs, printers, scanners and other types of media devices or storage devices. SCSI can connect multiple devices to a single adaptor (e.g., a host bus adapter (HBA) or port) on a SCSI bus. SCSI devices transfer bits in parallel and can operate in either asynchronous or synchronous modes.
All devices communicate in one of two ways; either one character at a time, or as a complete set of data in a predefined block. Therefore, SCSI devices can be classified as being either “character mode” or “block mode”. Terminals, printers, and modems are character devices since they handle data one character at a time. Hard drives and tape drives use blocks to transmit data as large chunks of information so they are block mode devices. Some SCSI devices function in either mode using two different device drivers.
A device driver is a software module that is part of the kernel and that supports access to a peripheral SCSI device. For example, in a Linux OS, when a SCSI device needs to be accessed, the Linux kernel passes the I/O request to the device driver, which handles the communication with the SCSI device. Special device files are used to access the SCSI devices which are connected to a Linux system such as printers, tape drives, terminals and disk drives that are used to store file systems. Special device files (hereafter device files) are kept in what is called a /dev directory and are given device file names representative of their function, such as “/dev/fd0” for a floppy drive, for example.
SCSI devices are identified by a device number. Device numbers allow the operating system to differentiate between the SCSI devices. The device number is constructed from a major number and a minor device number. In a standard Linux implementation, each SCSI device is assigned a major number and a minor number. The major number identifies the type of SCSI device (really the device driver), while the minor number represents an instance of a SCSI device of the type specified by the major device number. For example, the device file name “/dev/fd0” identifies the first floppy drive on a system, with “fd” representing the major number, and “0” representing the minor number of the device file name. The major number in Linux is a number ranging from 0 to 255 that identifies the SCSI device type. The minor number is also a number ranging from 0 to 255 which specifies a specific SCSI device among a group of SCSI devices that share the major number.
SCSI devices and device drivers having been briefly introduced, a discussion of the Linux SCSI I/O Subsystem including driver architecture, SCSI device addressing within the Linux SCSI I/O Subsystem and initialization of the Linux SCSI I/O Subsystem is provided, in order to further describe a background context for the present invention.
Linux SCSI I/O Subsystem
The Linux SCSI I/O subsystem is made up of two basic elements, each of which may be represented by data structures. The first basic element is a SCSI host. A SCSI host is a physical piece of hardware, such as a SCSI controller or HBA. If a Linux host has more than one SCSI controller of the same type, each instance will be represented by a separate SCSI host. This means that a device driver may control more than one instance of its controller. SCSI hosts are almost always the “initiator” of SCSI commands, thus a SCSI host may be termed a SCSI initiator device, with the host controller or HBA being an “initiator”. A scsi_host data structure is used to describe each instance of a host adapter (HBA) on the Linux system. The second basic element within the Linux SCSI I/O subsystem is a SCSI device. As discussed above, there are several types of SCSI devices: disk devices, tape devices, CD-ROM devices and generic devices. The different disk types have different major device numbers, allowing Linux to direct block device requests to the appropriate SCSI device type.
A scsi_device_structure is a data structure that is used to describe a single SCSI device. There will be a single instance of the structure allocated for each physical SCSI device that is detected. Any SCSI device (such as a CD changer) that responds to multiple LUNs (logical unit numbers, a type of identifier discussed further below) will be treated as separate instances of independent SCSI devices. The scsi_host and scsi_device_structure(s), which are objects, are allocated at the time the SCSI bus is scanned by a mid-level SCSI layer. Objects, as are known in the computer literature, represent architectural abstractions or containers that encapsulate data types, data structures, services or other objects that are related in some way.
Driver Architecture Overview.
FIG. 1 illustrates a driver architecture for the Linux SCSI I/O Subsystem. The Linux SCSI I/O subsystem (hereinafter SCSI subsystem 1000) has a three (3) level driver architecture with the “upper” level layer 100 being closest to a user space/kernel space interface 110, while the “lower” level layer 150 is closest to hardware (e.g., physical SCSI device, not shown). As will be explained further below, the upper level drivers (ULDs) in upper level 100 are commonly known by a terse two letter abbreviation (e.g., “sd” for SCSI disk driver). The names of the corresponding module drivers which, for historical reasons, sometimes differ from the built-in driver names are shown in brackets in FIG. 1.
The ULDs in upper level layer 100 maintain the kernel side of the Linux OS's user-kernel interface for the logical class of SCSI devices they represent (e.g., disks). They are also responsible for managing certain kernel and SCSI subsystem 1000 resources such as kernel memory and SCSI command structures. Applications in the user space access these ULDs by opening a device file (block or char) typically found in the /dev directory tree.
Accordingly, the upper level layer 100 supports the kernel/user interface. For example, in the case of sd (disk) drivers and sr (CD-ROM) drivers, the kernel/user interface is a block device interface, while for st (tape) drivers and sg (generic) drivers, the kernel/user interface is a character device interface. Any operation using the SCSI subsystem 1000 (e.g., reading a sector from a disk) involves one driver at each of the 3 levels, e.g., sd drivers at upper level 100, SCSI mid-level driver at SCSI mid level layer 125 and an HBA driver of lower level layer 150.
The mid-level SCSI layer 125 is common to all operations. The mid-level SCSI layer 125 defines internal interfaces and provides common services to the upper and lower level drivers. IOCTLs (input/output control commands) provided by the mid-level SCSI 125 layer are available to file descriptors belonging to any of the four ULDs (sd, sr, st, sg device drivers).
The most common operation on a block SCSI device is to “mount” a file system. For an sd device, typically only a partition is mounted (e.g., mount -t ext2/dev/sda6/home). For an sr device, usually the whole device is mounted (e.g., mount -t iso9660/dev/sr0/mnt/cdrom). A dd command can be used to read or write from block SCSI devices. In this case the block size argument (“bs”) needs to be set to the block size of the device (e.g., 512 bytes for most disks) or an integral multiple of that device block size (e.g., 8192 bytes). A recent addition to the block subsystem allows a SCSI device (or partition) to be mounted more than once, at different mount points.
Sd is a member of the generic disk family, as is the hd device from the IDE (Integrated Device Electronics, a mass storage device interface standard) subsystem. Apart from mounting sd devices, an fdisk command is available to view or modify a disk's partition table. Sr is a member of the CD-ROM subsystem. Apart from mounting file systems (e.g., iso9660), audio CDs can also be read. The latter action does not involve mounting a file system, but typically involves invoking some IOCTLs. St is a char device for reading and writing tapes. Sg is a SCSI command pass through device that uses a char device interface.
SCSI Addressing.
Linux has a four level hierarchical addressing scheme for SCSI devices: (a) SCSI adapter number [host]; (b) channel number [bus]; (c) ID number [target]; and (d) LUN [lun]. The term “LUN” is a common SCSI abbreviation of Logical Unit Number. The LUN identifies the logical unit-within a SCSI target device. A SCSI target device is a SCSI device containing logical units that service commands from a SCSI initiator device in order to carry out a specified task for example. As is known, a logical unit may be defined as addressable blocks of storage created from one or more disks (also called spindles) contained within a SCSI device. A logical unit may provide a unique connection to an application program or another SCSI device. “Bus” is used herein in preference to “channel” in the description below.
The SCSI adapter number is typically an arbitrary numbering of adapter cards on internal I/O SCSI buses (e.g., PCI, PCMCIA, ISA, etc.) of a computer (Linux host) running the Linux OS. Such adapters are commonly referred to as host bus adapters (HBAs) or “SCSI hosts”. HBA numbers are issued by the Linux kernel in ascending order, starting with 0. Each HBA may control one of more SCSI buses.
Each SCSI bus can have multiple SCSI devices connected to it. In SCSI parlance, the HBA may be called the “initiator” (e.g., it may be an HBA of a SCSI initiator device) and takes up one SCSI ID number (typically the number “7”). The initiator talks to targets, commonly also known as SCSI target devices (e.g., disks), as briefly discussed above.
On SCSI parallel buses, the number of SCSI IDs are related to the width of the bus. For example, 8-bit buses (sometimes called “narrow” SCSI buses) may have 8 SCSI IDs, of which one ID is taken by the HBA, leaving 7 IDs for SCSI devices. Wide SCSI buses are 16 bits wide and can have a maximum of 15 SCSI devices (e.g., targets) attached. The SCSI-3 draft standard allows a large number of IDs to be present on a SCSI bus, with each SCSI device adapted to contain multiple (LUNs). These multiple LUNs are typically used by sophisticated tape and CD-ROM units that support multiple media. Accordingly, Linux's flavor of SCSI addressing is a four level hierarchy, namely <scs(_adapter_number), channel, id, lun>, which is most commonly known and denoted with <host, bus, target, lun> encoding, i.e., (h0b0t0l0). Thus, a path from a SCSI host to a SCSI device, or to a logical unit represented by a LUN on a SCSI device, can be determined from the <host, bus, target, lun> or (h, b, t, l) address of a logical unit.
Storage area networking is predicated on the replacement of parallel SCSI transport with networked storage SCSI devices and tape SCSI devices behind the server or Linux Host. The vast majority of storage area networks (SANs) use a Fibre Channel (FC) medium for the underlying network transport and, at the upper layer, move data to and from disks with serial SCSI protocol, as described in the current draft SCSI Primary Commands-3 (SPC-3) document. The latest version of the SPC-3 document, found at ftp://ftp.t10.org/t10/drafts/spc3/spc3r09.pdf, provides SCSI primary command standards that are designed to be used industry wide. The combination of a high-speed transport (FC) with native SCSI protocol results in an efficient means to deploy servers, disk arrays, and tape subsystems and frees all components from the constraints of parallel SCSI architecture.
The SCSI-3 device driver supplied by the HBA vendor is responsible for mapping FC storage resources to the bus, target, lun conventions required by Linux. Since FC addresses are self configuring, the mapping between port addresses, (which may change) and the upper level SCSI device driver designations are maintained by the HBA and the HBA's device driver interface to Linux.
Initializing the SCSI Subsystem
FIG. 2 is a sequence diagram according to unified modeling language (UML) principles that describes initializing a Linux SCSI I/O Subsystem (“SCSI subsystem”) in order to detect one or more SCSI devices. FIG. 2 is provided in order to illustrate how device-driver-generated device file names are created by the standard Linux (and UNIX) implementations. FIG. 2 is also intended to illustrate that in standard Linux (and UNIX), there is currently no mechanism or method, in standard Linux, by which device driver-generated device file names may be used to determine multiple, physical (communication) paths, also called SCSI paths, from a Linux host consisting of multiple HBA adapter cards (also know simply as HBAs or SCSI hosts) connected to a single SCSI device (SCSI target device) or single logical unit within a SCSI target device.
Sequence 200 in FIG. 2 depicts the various interactions between a mid-level SCSI driver 102, an HBA driver 104, a SCSI device (e.g., SCSI device) 106, SCSI device drivers 108 (e.g., ULDs sd, st, sr, sg), a Linux kernel 110 and a file system 112. SCSI device 106 may represent one or a plurality of SCSI devices, more particularly one or more logical units (represented by their LUNs) on a SCSI device.
An initial assumption to be made is that for the Linux host to recognize any newly added SCSI devices connected to the host, the Linux host needs to be rebooted. This is shown in FIG. 2 When a new SCSI device is connected to a Linux host, a re-boot of the Linux host needs to be initiated by a user of the Linux host. This is shown in FIG. 2 by an asynchronous procedure call (202) to the Linux Subsystem. During the reboot, kernel 110 makes a synchronous procedure call 204 (depicted by solid two-sided arrow) to mid-level SCSI driver 102, to initialize the SCSI subsystem. Mid-level SCSI has the job of running through all of the low-level drivers (HBA drivers 104) that were compiled into the system, and calling their detect ( ) function to see if there are any physical adapter cards present in the system which any of the HBA drivers 104 can control.
In general, Linux finds out which of the SCSI hosts (HBAs), or controllers, that were built into the kernel 110 at kernel build time have hardware to control. Each built-in SCSI host has a scsi_host_template entry in what is called builtin_scsi_hosts vector. This vector is an array of all the adapter cards that the kernel 110 knows. A scsi_host_template data structure contains pointers to routines that carry out SCSI host specific actions such as detecting what SCSI devices are attached to this SCSI host. These routines (detect( ), release( ), ioctl( ), command( ), queuecommand( ), abort( ), reset( ), proc_info( ) etc.) are called by the SCSI Subsystem 1000 as it configures itself and they are part of the HBA driver 104 supporting this host type.
Referring to FIG. 2, the mid-level SCSI driver 102 has the job of running through all of the low-level HBAs that are built into the kernel 110 at time of reboot, and calls a detect( ) routine 206 (shown as a synchronous procedure call) to each HBA driver 104. This is done to see whether a HBA driver 104 has found adaptor cards, and hence an HBA. In other words, the detect entry point for an HBA driver 104 is called. For each HBA recognized (see return 208) by an HBA driver 104, a scsi_host data structure is created (see message 212). This is created for each newly detected HBA during the detect( ) routine by calling a function called scsi_register( ) in the mid-level scsi driver 102.
Information about the SCSI device drivers 108 (e.g., ULDs) are maintained in a scsi_device_template data structure. At kernel 110 bring-up time (e.g., during re-boot), a scsi_device_template for each known device drivers 108 is loaded (210) by the kernel 110. This structure, among other things, contains pointers to the detect( ), init( ), attach( ), finish( ) and detach( ) functions, which are driver routines performed by SCSI device drivers 108. The scsi_device_template is used to describe a single type of device driver 108. In the current implementation of the SCSI subsystem, there is an absolute maximum of four (4) instances of this structure on any given system; one for sd driver, one for CD-ROMs (sr driver), one for tapes (st driver), and one for the generic pass-through devices (sg driver). If other driver types are added (e.g., scanner), a new scsi_device_template is created. A single physical SCSI device can be attached to more than one scsi_device_template (usually this is a generic driver, plus another device driver (e.g., an sd, sr, st driver). The mid-level SCSI driver 102 will eventually use the registered scsi_device_template of the SCSI device drivers 108 to invoke the SCSI device drivers 108, e.g., the scsi_device_template is called first to invoke certain driver functions or routines such as detect( ), attach( ), detach( ), finish( ), etc.
Each detected SCSI host for which there are real SCSI devices attached has its scsi_host_template data structure added to a scsi_hosts list of active SCSI hosts. Each instance of a detected host type is represented by a scsi_host data structure held in the scsi_hostlist list. For example, a system with two PCI SCSI controllers would have two scsi_host entries in the list, one per controller. Each scsi_host points at the scsi_host_template representing its SCSI device driver 108.
Turning now to FIG. 2, for each HBA driver 104 that recognizes an HBA (SCSI host), the mid-level SCSI driver 102 adds, by internal function 212, the HBA driver 104's Scsi_Host data structure (as mentioned above), which are stored in internal memory, to a scsi_hosts list of active SCSI hosts. For each recognized HBA instance, a sScsi_host_list data structure is created by internal function 214.
B. Discovering SCSI devices.
Now that every SCSI host (HBA) has been discovered, the SCSI subsystem 1000 must find out what SCSI devices are attached to each host's bus. SCSI devices are numbered between 0 and 7 inclusively, each SCSI device's number or SCSI identifier being unique on the SCSI bus to which it is attached. SCSI identifiers are usually set by jumpers on the SCSI device. This is done in a bus scan.
A bus scan is where commands are sent out to each SCSI host (HBA) on the system to inquire as to what SCSI devices, if any, are present on each bus. The general theory is simple: loop over all SCSI ID's, over all of the channels, and see what SCSI devices 106 are out there connected to the system. This is done by sending a SCSI command TEST_UNIT_READY.
Referring to FIG. 2, mid-level SCSI driver 102 makes a procedure call to SCSI devices 106. This is done by first sending TEST_UNIT_READY (TUR) command 216. Any response (see return 218) which indicates the presence of an actual SCSI device 106 is taken as an affirmative by mid-level SCSI driver 102. Another procedure call is made to discovered SCSI devices 106. When a SCSI device 106 responds, its identification is read by sending it a command. This is a SCSI INQUIRY command 220, that requests certain identifying data from SCSI device 106 (which could be a single LUN SCSI device or component logical units (as represented by multiple LUNs) of a SCSI device. The identifying data is sent to the mid-level SCSI driver 102.
For each SCSI device 106 that responds, the retrieved information is used in order to create (222) a scsi_device data structure that points to the scsi_host. A scsi_device data structure is a data structure that includes host#, bus#, target# (SCSI ID), and LUN fields, (e.g., the h,b,t,l address) to a SCSI device 106, such as a LUN (logical unit). The scsi_device data structure may also include fields which gives Linux the vendor's name, the SCSI device's model name and any revision names, for example. The scsi_device data structure has a corresponding SCSI device pointer (SDP) which points to the scsi_host. There is a SDP for each SCSI device. The SDP contains fields which reference an h,b,t,l of the scsi_device data structure and hence a SCSI device.
Additionally for each discovered SCSI device 106, mid-level SCSI driver 102 makes a procedure call to see what ULDs might be willing to drive the SCSI device 106. For example, more than one device driver 108 (e.g., an sd and sg driver, or st and sg driver) may each want to “claim” a discovered SCSI device 106. This is accomplished by calling the detect ( ) entry point (224) for all of the registered ULDs using the scsi_device_template structure for each SCSI device driver (e.g., registered at 210). In FIG. 2, there is a box 226 that encompasses calls/functions 202–224. This indicates that a looping function is performed, e.g., these steps are repeated for each HBA built into kernel 110.
C. Calling SCSI device Driver Routines
SCSI commands (e.g., a SCSI INQUIRY command) are represented by a scsi_cmnd data structure and are passed to the device driver 108 by calling device driver routines within its scsi_host_template data structure. Every SCSI device 106 that is found is represented by a scsi_device_structure, each of which points to its parent scsi_host. All of the scsi_device_structure(s) are added to a scsi_devices list.
Each “SCSI type” is represented by a scsi_type_template data structure. This contains information about this type of SCSI device and the addresses of routines to perform various tasks. The SCSI subsystem uses these templates to call various SCSI type device driver routines (detector init( ), finish( ), attach( ), detach( ) etc.) for each type of SCSI device 106.
For example, an init( ) entry point routine (228) is called for all ULDs (SCSI device drivers 108) for initialization; this will allocate internal data structures. For all discovered SCSI devices 106, an attach( ) entry point routine (232) is called to insert a pointer to the scsi_device data structure of each discovered SCSI device 106 (again, loop 216) into the internal tables of the ULD (SCSI device driver 108). In other words, if the SCSI subsystem wishes to attach a SCSI disk device, it will call the SCSI disk type attach routine. The scsi_type_template data structures are added 236 (to the scsi_device_list) if one or more SCSI devices 106 of that type have been detected. Block 234 indicates that calling of the attach( ) routine is repeated for all discovered SCSI devices 106.
The final phase of the SCSI subsystem initialization is to call finish( ) routines 238 for each registered scsi_device_template. For the SCSI disk type (sd), this routine spins up all of the SCSI disks that were found and then records their disk geometry. It also adds the gendisk data structure (not shown) representing all SCSI disks to a linked list of disks. Once Linux has initialized the SCSI subsystem, the SCSI devices 106 may be accessed.
In order to access SCSI devices 106, e.g., through device-driver-generated device file names in file system 112, which are required in order to talk to a SCSI device 106, device nodes need to be created for the SCSI devices 106. These device nodes need to be created by a system administrator on the Linux host, for example. In Linux a MAKEDEV utility 114 creates all these device nodes for various device drivers 108. As is known, MAKEDEV is a user mode utility run from boot-up scripts.
The MAKEDEV utility generates (see message 240) all device files required to talk to a SCSI device 106. However, it creates entries for a list of all conceivable SCSI devices in file system 112 (call a/dev/directory), most of which may not be actually be present in the system. This means that the /dev/directory does not truly represent what SCSI devices 106 are actually connected to the system.
In standard Linux (and also in UNIX), there is no way to determine multiple paths to the same logical unit (LUN) of a SCSI device 106 using the device-driver-generated device file names. Moreover, a SCSI host may have several physical paths (SCSI paths) to a single logical unit, as represented by its LUN) on a SCSI device.
Each physical path to a SCSI device 106 (or to a LUN behind a logical unit 106) results in a separate device file name. Most of the device-driver-generated device file names (e.g., /dev/sda, /dev/sgl, etc.) have h,b,t,l mapping in them, as they are associated with a minor number. So it is possible that a user (e.g., application in the user space) by mistake can mount two different device files pertaining to the multiple paths to the same SCSI device. For example, since the SCSI disk device is a block device, attempting to mount multiple device files on a single SCSI disk device could lead to a file system corruption. In a SAN configuration, which contains an enormous amount of SCSI devices (and hence a large number of device files), the probability of such a corruption is too high.
Currently, fault tolerant drivers, which try to avoid a single point of failure, such as by using multiple HBA's on each Linux Host, for example, normally mark one of the multiple paths to a SCSI device as active. All the other paths to the same SCSI device are marked as passive. Whenever there is a problem, the driver attempts to “fail over” from the active path to one of the passive paths. So, fault tolerant drivers need an effective way to identify the multiple physical SCSI paths to the same SCSI device. What is needed is a mechanism or technique to make it possible to detect multiple physical SCSI paths to the same SCSI device or logical unit.